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Objectives: Although evidence-based medicine (EBM) is a useful method for integrating evidence into the decision-making pro- 
cess of occupational physicians, occupational physicians lack EBM knowledge and skills, and do not have the time to learn the 
EBM method. In order to enable them to educate themselves at the time and place they prefer, we designed an electronic EBM 
course. We studied the feasibility and utility of the course as well as its effectiveness in increasing EBM knowledge, skills, and be- 
haviour. 

Methods: Occupational physicians from various countries were included in a within-subjects study. Measurements were conduct- 
ed on participants' EBM knowledge, skills, behaviour, and determinants of behaviour at baseline, directly after finishing the course 
and 2 months later (n = 36). The feasibility and utility of the course were evaluated directly after the course (n = 42). 
Results: The course is applicable as an introductory course on EBM for occupational physicians in various countries. The course is 
effective in enhancing EBM knowledge and self-efficacy in practising EBM. No significant effect was found on EBM skills, behav- 
iour, and determinants of behaviour. After the course, more occupational physicians use the international journals to solve a case. 
Conclusion: An electronic introductory EBM course is suitable for occupational physicians. Although it is an effective method for 
increasing EBM knowledge, it does not seem effective in improving skills and behaviour. We recommend integrating e-learning 
courses with blended learning, where it can be used side by side with other educational methods that are effective in changing 
behaviour. 

Keywords: Evidence-based medicine, Distance education, Occupational health, Medical education 



Introduction 

Decisions in occupational health care by occupational physi- 
cians (OPs) should be based on professional competences of 
the OP, workers' (patients'), and employers' preferences, and 
evidence from research information [1]. In order to accomplish 
this, OPs can benefit from integrating the evidence-based medi- 
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cine (EBM) method into their daily practice. When applying 
EBM as described by Sackett et al. (1999) [2], OPs should start 
with formulating an answerable question, then searching and 
acquiring relevant literature, appraising it for quality, applying 
the findings, and evaluating the performance of the decision. 
In doing so, OPs can use evidence from occupational cohort 
and randomized controlled studies, for example, but they can 
also use evidence stemming from studies performed by a wide 
variety of medical and other disciplines such as occupational 
hygiene and psychology [3]. In contrast with the occupational 
setting, EBM is commonly used in the clinical setting and has 
been proven to enhance the quality of health care [4-6]. 

Over the last few years, there have been first attempts 
to promote EBM in the occupational and public health care 
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setting. Associated evaluation studies showed that there are 
various opportunities to enhance the use of EBM [1,7-10]. As 
a result, we already know that OPs are often unaware of the 
high volume of relevant questions that can be derived from 
their own daily practice and of the opportunities to answer 
most questions when following EBM strategies [11]. Moreover, 
when answering their information demands, OPs rely rather on 
their own experience or on advice from colleagues and 'experts' 
instead of searching for evidence in the scientific literature 
themselves [12]. Interestingly, it has been shown that the advice 
OPs received from 'experts', including colleagues, do not reflect 
the best evidence from the literature [9]. 

An evaluation of an EBM course combined with recur- 
rent peer group sessions showed that EBM knowledge, skills, 
and attitude of OPs improved significantly, as well as the qual- 
ity of therapy advice of OPs in the short term [13,14]. The par- 
ticipating OPs of that study valued EBM as a useful method for 
enhancing professional performance, although they thought the 
combination of the course and the peer group sessions were in- 
tensive and time-consuming. In many other studies evaluating 
EBM strategies in settings other than the occupational health 
field, time constraints are also reported as a main barrier for the 
uptake of EBM [15-18]. 

In particular, to overcome this barrier of limited time and 
to promote EBM among OPs internationally, we designed an 
EBM e-learning introductory course in the English language. 
The aim was to enable OPs to train themselves in the basics of 
EBM at the time and place of their choice, determining their 
own pace and repeating the course whenever they choose [19]. 
However, although e-learning or web-based education is be- 
ing used increasingly, little is known about the feasibility and 
the effectiveness of it when used by OPs in a wide variety of 
countries. One study on the preferences of OPs on distance 
learning reveals that they prefer online practice in addition to 
printed education material [20] . Another recent study shows 
that e-learning is just as effective as lecture-based learning in 
enhancing OPs' knowledge on mental health care [21]. One 
interesting question deals with the suitability of e-learning 
developed in one country or region, for the daily practice and 
learning demands of occupational professionals across various 
countries. We expect these professionals to have a large variety 
of professional activities and backgrounds in vocational train- 
ing. Barriers to effective implementation might be expected in 
technical problems, such as access to the internet and availabil- 
ity of scientific literature, and in the wide variety of legislation, 
cultures, and languages [22,23]. All of these questions and un- 
certainties demonstrate a need for evaluation in order to learn 
for the future. 



Therefore, in this study, we first decided to evaluate 
whether the EBM e-learning course is applicable for OPs in 
different countries, in terms of feasibility and utility. Next, we 
wanted to know whether the course enhances the EBM knowl- 
edge and skills of OPs. Finally, in order to determine whether 
the EBM e-learning course is capable of changing the EBM be- 
haviour of OPs, we used the attitude, social influence, and self- 
efficacy (ASE) model [24]. This model is often used in the field 
of health education. Taken together, attitude, social influence, 
and self-efficacy determine the motivation for certain human 
behaviour, the intention. The ASE model implies that intention 
predicts behaviour [25]. In accordance with the ASE model, 
social influence, perceived self-efficacy, and the intention to 
practice EBM as determinants of actual EBM behaviour were 
included as variables in the study. 

Materials and Methods 

Study population and recruitment procedure 
Study participants were OPs from various countries participat- 
ing in a local postgraduate course in the field of occupational 
health care organized by experienced (postgraduate) education 
and training centres. To approach these centres, national and 
international contacts of our institute (Coronel Institute of Oc- 
cupational Health) as well as a list of the members of the Eu- 
ropean Association of Schools of Occupational Medicine were 
used. 

The education centres received information about our 
study via email in June and were invited to assist in our study 
in a four-month period from August until November 2007. If 
the centre agreed to assist, they were asked to provide the email 
addresses of OPs who were potential course participants of 
the centre. These potential participants received information 
on the EBM e-learning evaluation study as well as an informed 
consent form via email. Only those course participants who 
sent back their signed informed consent form were included in 
our study. Participants were requested to complete the EBM 
e-learning course in their leisure time. If they were by chance 
planning to attend an EBM course or a similar course offered 
by their educational centre, they were asked to follow our elec- 
tronic EBM course first. Some educational centres offered our 
electronic EBM course as part of their own postgraduate pro- 
gramme. When this was the case, the electronic EBM course 
was provided at the start. 

EBM e-learning course 

The format of the EBM e-learning course (http://www.nspoh- 
on-line.nl./ebm-who/) was based on an effective e-learning 
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course on mental health care for OPs, and its content was based 
on the recently published first international practical guide on 
EBM for occupational health [3]. The course is in English, lasts 
about one hour and a half, and comprises seven topics. The 
seven topics are: Basic principles of EBM (23 screens); types of 
questions (26 screens); the patient, intervention, comparison, 
outcome (PICO) method (10 screens); information sources and 
search strategies (20 screens); searching in MEDLINE with 
PubMed (29 screens); evaluating results (41 screens); answer- 
ing questions (15 screens); in a total of 164 screens. Every topic 
starts with a screen on the issues that will be covered in that sec- 
tion (Appendix). Then, screens with information are alternated 
with screens with assignments. Feedback on the assignments is 
provided directly. Figs 1, 2, and 3 are examples of the screens. 
Furthermore, the course has an online library available where 
participants can find: direct access to PubMed; an overview of 
relevant keywords (search terms) for occupational health pro- 
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Fig. 1 . 'Start' screen on the topic "Evaluating results" presents the issues 
covered. 



EBM E-learning: Feasible for Occupational Physicians 

Saf Health Work 2012;5:199-208 

fessionals; an overview of relevant databases; relevant articles 
and books; an overview of help files; examples of critically ap- 
praised topics; and links to websites that support professionals 
in the field of EBM. 

Study design and type and timing of measurements 
A within-subjects study design with pre- and post-test evalua- 
tion was used. Pre- and post-test measurements were conduct- 
ed by means of an electronic questionnaire on the participants' 
knowledge on, skills in, and attitude towards EBM, as well as 
EBM behaviour and determinants of that behaviour. Partici- 
pants received the questionnaire at 3 time points. After provid- 
ing their informed consent, they received a baseline question- 
naire (TO) by email. As soon as the participants completed this 
questionnaire, the URL of the electronic EBM course and in- 
structions on how to complete the course were provided. Upon 
completion of the course, the second questionnaire (Tl) was 
automatically presented in a pop-up screen and the participants 
had to immediately fill in this questionnaire. Approximately 2 
months after completion of the electronic EBM course and the 
second questionnaire, the last questionnaire (T2) was sent to 
the participants by email. 

Outcome 

In order to describe the study population, demographics of 
participants (country of residence, age, gender), experience as 
an OP, work setting, and previous training in research methods, 
epidemiology, statistics, or EBM were assessed at TO. The util- 
ity and feasibility of the electronic EBM course were assessed 
at Tl, directly after completing the course. Knowledge, skills, 
attitude, behaviour, and determinants of behaviour related to 
EBM were assessed at all moments TO, Tl, and T2. Evidence 
use was only assessed at TO and T2, since Tl was too soon af- 
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ter TO to expect any change on evidence use. 

Feasibility and utility of the electronic EBM course 
Feasibility was defined as the extent to which the EBM elec- 
tronic course could be completed by the participants. Utility 
was defined as the extent to which the participants perceived 
the content of the EBM electronic course as useful for the 
improvement of their performance. This means that we evalu- 
ated to what extent the course was clear in both form and 
content, and thereby to what extent it was attuned to the prior 
knowledge and to the practice setting of the participants. The 
researchers formulated 14 statements to which the participants 
could agree on a 4-point Likert scale; scores ranged from 1 
(strongly disagree) to 4 (strongly agree). 

EBM knowledge 

EBM knowledge was defined as the knowledge partici- 
pants had on the basic principles of EBM, how to search in 
PubMed/MEDLINE, and how to appraise the literature criti- 
cally. To measure the participants' knowledge, we used the criti- 
cal appraisal part of the questionnaire of Taylor et al. [26,27], 
combined with questions designed by one co-author (F. V.) on 
the basic principles of knowledge on EBM and searching abili- 
ties in Pubmed. Our combined list consisted of 31 true/false/ 
don't know questions. A correct, incorrect, and 'don't know' re- 
sponse yielded a score of +1,-1, and 0 points, respectively, and 
were added up to form an overall sum score. The sum scores 
were subsequently converted into scores between 0 and 100 
in order to facilitate standardisation on all outcome measure- 
ments. 

EBMskills 

EBM skills were defined as the ability to perform the steps of 
EBM described by Sackett et al. [28] (1996): formulate a clear 
cfinical question from a patient's problem; search the literature 
for relevant clinical articles; evaluate the evidence for validity 
and usefulness; and implement the useful findings in practice. 
For the measurement of EBM skills, we used the skills part 
of the Fresno test used by Schaafsma [14] and adapted the 
scenarios to occupational health-related situations familiar for 
professionals in many countries. The test consisted of eight 
open questions; we used the standardised grading system of the 
Fresno Test [6,29]. Again, the sum scores were converted into 
scores between 0 and 100. 

EBM attitude, behaviour, and determinants of behaviour 
EBM attitude was understood to represent the beliefs, feelings, 
and values of the participants towards EBM. In accordance 



with the ASE-model, we included the social influence related 
to EBM, the perceived self-efficacy to practice EBM, and the 
intention to practice EBM in the study [23]. The social context 
can be described as the processes whereby OPs' thoughts, feel- 
ings, and actions on EBM are directly or indirectly being influ- 
enced by others. Self-efficacy can be seen as the OPs' belief in 
his or her ability to perform EBM practice. The five constructs 
of the ASE model were measured by means of the question- 
naire from Schaafsma et al. [12], including 22 statements. In 
the original Dutch questionnaire, there were five statements on 
the use of evidence-based national guidelines for OPs. Since 
guidelines are non-existent in some countries, three statements 
were removed from the original questionnaire and two state- 
ments were adapted into statements on the use of EBM in 
general. This resulted in: 1) 4 items measuring attitude towards 
EBM (e.g., "using EBM in daily practice improves the quality 
of the physician"); 2) 5 items measuring the social context of 
the OP, reflecting the support received from the social work 
environment such as from supervisors or colleagues; 3) 5 items 
measuring self-efficacy including the feeling of self-confidence 
towards practising EBM; 4) 3 items measuring the intention to 
use EBM in the near future; and 5) 2 items measuring actual 
EBM practice. Our instrument used a 5-point Likert scale: 
strongly agreeing with a 'positive' statement and strongly dis- 
agreeing with a 'negative' statement scored '5' points, strongly 
disagreeing with a positive statement or strongly agreeing with 
a negative statement scored ' 1 ' point. For each of the 5 vari- 
ables, a sum score was calculated to obtain an overall score. 
Again, the sum scores were converted into scores between 0 
and 100 in order to facilitate standardisation on all outcome 
measurements. 

Evidence use 

We used a broad concept of evidence use, defined as the self- 
reported time spent on keeping up-to-date and solving a spe- 
cific case and, in addition, the use of a number of sources by 
participants when doing so. To assess this, we used the 'Reading 
and evidence seeking behaviour' part of the questionnaire of 
Taylor et al. [27]. First, the OPs reported the number of hours 
spent on solving a specific case and on keeping up-to-date over 
the previous month. In addition, they reported the number of 
articles read to solve a specific case and the number of articles 
read to keep up-to-date during the previous month. Second, 
they were asked to report the proportion (%) of these articles 
that they read thoroughly, skimmed, or read only the abstract 
of. Finally, the OPs reported how often they used several kinds 
of sources to keep up-to-date and to solve their cases. The 
sources asked for were: review articles in international journals, 
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original research reports in international journals, national 
journals, textbooks, Internet resources /computer databases or 
similar, guidelines, the Cochrane Library, and colleagues. 

Statistical analysis 

Proportions (means and standard deviation [SD]) of our mea- 
sures of the utility and feasibility of the electronic EBM course 
were calculated. To determine if the electronic EBM course 
had an effect on EBM knowledge, skills, attitude, and deter- 
minants of EBM behaviour over time, the total scores on TO, 
Tl, and T2 were calculated and analysed by means of general 
linear model for repeated measures. When trends were found, 
a paired t-test was used post-hoc to analyse differences in scores 
between TO and Tl. Mean scores (SD) and proportions of the 
items on evidence use were calculated. Statistical analyses were 
carried out using SPSS version 13.0 (SPSS Inc., Chicago, IL, 
USA). 

Results 

Baseline characteristics 

Overall, 102 OPs from 16 countries returned the informed con- 
sent form and 84 OPs filled in a baseline questionnaire (TO). 
Directly after conducting the electronic EBM course (Tl), 42 
OPs returned a completed questionnaire. Approximately two 
months after completing the electronic EBM course, 36 OPs 
returned the last questionnaire (T2). Table 1 shows the personal 
characteristics of the participants at baseline and T2. 

The majority of participating OPs are from Europe, 
mainly Belgium (n = 13 at TO; n = 0 at T2), the Czech Repub- 
lic (n = 9 at TO; n = 6 at T2), and Italy (n = 8 at TO; n = 8 at 
T2); Africa, mainly South Africa (n = 1 1 at TO; n = 5 at T2); 
and the West Pacific, mainly Japan (n = 16 at TO; n = 4 at T2). 
Almost 60% of the OPs are less than 40 years of age and about 
half of the OPs are still in training. The OPs mostly work at 
an occupational health service provider or work in a company, 
and almost 80% have had training in methodology, statistics, 
or EBM. No major differences of characteristics can be distin- 
guished between the participants at baseline and the remaining 
group at T2. 

Utility and feasibility of the electronic EBM course 
Table 2 shows that the vast majority of the OPs agree with the 
positive statements on the utility and feasibility of the course. 
The highest agreement is on the statements that the course of- 
fered sufficient information for an introductory course (100%), 
that it helped them to better practice EBM (97.6%), and that it 
improved their quality of work (97.6%). The relatively smallest 
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Table 1. Baseline characteristics of the participating OPs 
Participants' characteristics Baseline (N = 84) T2 (N = 36) 



WHO regions 

Europe 46(55) 22 (61) 

Africa 14(17) 6(17) 

The Americas 1(1) 1(3) 

South-East Asia 3 (4) 0 

The West Pacific 20 (24) 7 (1 9) 

Gender 

Female 42(50) 19(53) 

Male 42(50) 17 (47) 
Age, years 

< 30 23 (27) 8 (22) 

30-39 30(36) 14(39) 

40-49 13(16) 5 (14) 

50-59 14(17) 8(22) 

>60 1(1) 0 

Missing 3 (4) 1 (3) 

OP 

Yes 27 (32) 14(39) 

Still in training 38 (45) 1 5 (42) 

Other* 14(17) 5(14) 

Missing 5 (6) 2 (6) 

Setting 1 , n 

OHS provider 29 7 

OHS clinic 10 6 

In company 18 6 

Alongside GP practice 4 2 

Hospital 12 6 

Other 23 10 
Former training in + 

Research methods 34 1 6 

Epidemiology 48 23 

Statistics 33 1 6 

EBM 20 8 

None 18 6 



Values are presented as number (%) or number only. 
OP: occupational physician, T2: the last questionnaire returned 
approximately two months after completing the electronic EBM 
course, WHO: World Health Organization, OHS: occupational health 
service, GP: general physician, EBM: evidence-based medicine. 
Occupational and public health expert, occupational safety and 
health expert, studying occupational medicine; + more than one an- 
swer is possible. 
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Table 2. Utility and feasibility of the course: percentage of OPs 



that (strongly) agree with the statements (n = 42) 

Statements Agree 

The objective of the online course was clear 39 (93) 

The content of the online course was understandable 38 (91 ) 

The online course met my expectations 39 (93) 

The online course offered me sufficient information for 42 (1 00) 
an introductory course 

The content of the online course matched my 33 (79) 
educational level 

I will be able to use what I learned in this online course 40 (95) 

The content of the online course fits to my work setting 34 (81 ) 

This online course will help me to better practice EBM 41 (98) 

This online course will help me to improve the quality of 41 (98) 
my work 

The difficulty level of the online course was appropriate 35 (83) 

The online course was presented in a clear logical manner 40 (95) 

The online format was a good way for me to learn EBM 38 (91 ) 

I enjoyed taking the online course 33 (79) 

I would like to take more of this kind of online course 39 (93) 
Values are presented as number (%). 



OP: occupational physician, EBM: evidence-based medicine. 

agreement is on the statements that the course matched their 
educational level and that they enjoyed taking the online course 
(both 79%). Ninety three percent of the participants would like 
to take more of this kind of online course. Overall, they appre- 
ciate the electronic EBM course with the mean figure of 7.6 on 
a 10-point scale. 

EBM knowledge, skills, attitude, behaviour, and 
determinants of behaviour 

Of the participants who returned their questionnaire at T2 (n = 
36), only 25 OPs filled in the skill part, 33 OPs the knowledge 
part, and 32 OPs the attitude, behaviour, and determinants of 
behaviour part of the questionnaire at all three time points. The 
electronic EBM course has an overall significant effect on the 
OPs' knowledge of EBM and their self-efficacy on practising it 
(p < 0.01 and p = 0.02, respectively). The overall effect of the 
intervention on the enhancement of EBM skills is not signifi- 
cant, but the skills enhance substantially at Tl and, although 
they decline, are still higher at T2 compared to TO. The initial 
high scores on attitude remain stable over time. Except for 
social context, all scores are higher after the course compared 



Table 3. OPs' EBM knowledge, skills, attitude, behaviour, and 
determinants of behaviour scores at TO, T1 , and T2 



Variables* 


n 




I U 




I 1 




I Z 


tbivi Knowiecigt? 


JJ 


DO 


/ (y.aj 


16 


U \ 1 z 




/4. / \ I I ./ ) 


EBM skills 


25 


35 


6(12.3) 


51 


0(15 


5) 


40.1 (12.3) 


EBM attitude 


32 


78 


1 (11.1) 


83 


1 (11 


2) 


80.6(13.0) 


Social context 


32 


65 


9 (15.3) 


65 


4(17 


1) 


66.6(19.2) 


Self-efficacy f 


32 


52 


2 (11.3) 


58 


1 (12 


0) 


59.2 (12.0) 


Intention to 
behaviour 


32 


71 


6(13.0) 


72 


5 (15 


2) 


72.2 (8.7) 


Behaviour 


32 


45 


6 (30.6) 


53 


8(31 


5) 


49.4 (30.5) 



Values are presented as mean (standard deviation). 
*Overall tests on trends at time of the intervention. 
f p < 0.05. 

OP: occupational physician, EBM: evidence-based medicine, TO: 
baseline questionnaire, T1: the second questionnaire returned di- 
rectly after conducting the electronic EBM course, T2: the last ques- 
tionnaire returned approximately two months after completing the 
electronic EBM course. 

with before, and the increases sustain at T2, but are not signifi- 
cant (TO versus Tl and T2, Table 3). 

Evidence use 

Table 4 demonstrates that the number of articles read and 
hours spent reading decreases after conducting the course, 
particularly on keeping up-to-date. The number of journal ar- 
ticles that OPs looked at or read thoroughly is, both before and 
after conducting the course, higher for keeping up-to-date than 
for solving a case. In addition, the proportion of articles from 
which the OPs read the abstract only, skimmed, or thoroughly 
read stays the same. 

At baseline, OPs particularly use the national and inter- 
national journals to keep up-to-date; they use textbooks and 
guidelines to solve a case. After conducting the electronic EBM 
course, particularly the proportion of OPs using Internet re- 
sources, colleagues, review articles, and guidelines for keeping 
up to date, increases. For solving a case, particularly the propor- 
tion of OPs who use review articles, original research reports, 
national journals, and internet resources, increases. 

Discussion 

The electronic EBM course is applicable as an introductory 
course on EBM for OPs in a wide variety of countries. Ninety 
percent of the participants agreed that the online format was 
a good way to learn EBM, and almost all participants agreed 
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Table 4. Evidence use of OPs (n = 36) before, and two months 
after conducting the electronic EBM course 



Number of journal articles looked at or 
read thoroughly in the last month 



To keep up-to-date 


7.9 ± 7.8 


5.6 ± 4.5 


To solve a case 


4.0 ± 6.6 


3.6 ± 6.2 


Number of hours spent reading 
professional literature in the last month 






To keep up-to-date 


15.5 ± 18.6 


11.1 ± 10.0 


To solve a case 


5.6 ± 8.0 


4.7 ± 4.5 


Proportion of the articles looked at 






Read thoroughly 


20 


20 


Skimmed 


25 


25 


Read abstract only 


50 


50 


Types of resource used to keep up to 
date 






International journals: review articles 


14(39) 


17 (47) 


International journals: original research 
reports 


13(36) 


14(39) 


National journals 


19(53) 


18(50) 


Textbooks 


17 (47) 


16(44) 


Internet resources/computer databases 
or similar 


23 (64) 


29 (81) 


Guidelines 


14(39) 


17 (47) 


The Cochrane Library 


2(6) 


3(8) 


Colleagues 


19(53) 


23 (64) 


Types of resource used to solve a case 






International journals: review articles 


14(39) 


18(50) 


International journals: original research 
reports 


7(19) 


14(39) 


National journals 


8(22) 


12(33) 


Textbooks 


18(50) 


18(50) 


Internet resources/computer databases 
or similar 


24 (67) 


29 (81) 


Guidelines 


21 (58) 


21 (58) 


The Cochrane Library 


2(6) 


4(11) 


Colleagues 


22 (61) 


23 (64) 



Values are presented as mean ± standard deviation, number (%), or 
percent only. 

OP: occupational physician, EBM: evidence-based medicine, TO: base- 
line questionnaire, T2: the last questionnaire returned approximately 
two months after completing the electronic EBM course. 



that the course helps to improve the quality of their work. The 
course is effective in enhancing the OPs' knowledge on EBM 
and the self-efficacy of OPs in practising EBM. No overall 
significant effect was found on EBM skills, behaviour, and de- 
terminants of behaviour, although a substantial enhancement 
of EBM skills was noticeable directly after the course. OPs 
tend to read fewer articles and spend fewer hours on reading to 
keep up-to-date after the course, compared to before. After the 
course, international journals are particularly used more often 
by a larger proportion of OPs to solve a case. 

To our knowledge, this study is the first research to test 
the applicability and effectiveness of an electronic EBM course 
on EBM knowledge, skills, and behaviour of OPs in a variety 
of countries. By means of the World Wide Web (WWW), 
we were able to include many OPs from all over the world in 
our study and test the electronic EBM course. However, we 
lost 57% of our participants after they completed the baseline 
questionnaire, which might have been the cause of the lack of 
change over time in parameters excluding EBM knowledge 
and self-efficacy. We also cannot exclude the fact that partici- 
pants who were discontent with the utility and feasibility of the 
course were more likely to drop out. 

Unfortunately, using WWW also raised some techni- 
cal problems which presumably contributed to the high drop- 
out rate. First of all, the participants' computers had to be de- 
activated for pop-up blockers and spam filters in order to open 
the webpage of the course. Some participants informed us that 
they were not able to accomplish this, and we could not provide 
technical support because of the distance barrier. Secondly, an 
attempt was made to hack our electronic EBM course, and for 
safety reasons as well as to restore all conditions and materials, 
the course was unavailable for several days. 

Another limitation of our study was the lack of a control 
group. The electronic EBM course has been developed as an 
activity embedded in the Global Network of World Health 
Organization (WHO) Collaborating Centers in Occupational 
Health and was freely accessible through the internet during 
the period of our study. This made it difficult to include a con- 
trol group. Compared to reference groups from others studies, 
the scores on skills in our study at baseline were relatively low 
[29]. The OPs' mean score on knowledge at baseline was some- 
what lower than the mean score of the control group of Taylor 
et al. [27]. This suggests that OPs in our study started with a 
relative lack of skills and knowledge in EBM. Attitude and 
determinants of behaviour at baseline were higher in this study 
compared to the randomized controlled trial (RCT) on the ef- 
fectiveness of a course organised in the Netherlands [13,14]. At 
follow-up, the scores were still higher in our group compared to 
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the scores of the Dutch intervention group. Presumably, only 
OPs who had a positive attitude towards EBM, participated 
in our study. The OPs in our study may be considered as 'in- 
novators' in the adaption of EBM, which is still a relatively 
new phenomenon in the occupational health field. Considering 
these facts, together with the small sample size, the results of 
this study lack generalisability to all OPs in the countries in- 
volved. 

Following other studies on this topic [27,30], evidence use 
was measured by self-assessment. As this method is susceptible 
to bias in terms of giving desirable answers by the participants, 
the increase of participants using international journals in solv- 
ing cases might be an overestimation. Measurement of changes 
in evidence use and EBM behaviour in fact require assessment 
in the practice setting [31]. 

Agius and Bagnall [32] evaluated the demands and merits 
of the use of WWW in (postgraduate) learning occupational 
health care. The study indicated that it can be a useful learning 
resource, since students valued the flexibility, suitability, ef- 
ficiency, and breadth of access to relevant information offered 
by WWW [32]. A recent RCT among Dutch OPs showed that 
e-learning was just as effective as didactic learning in enhanc- 
ing knowledge [21]. There are no other studies on e-learning 
among postgraduate OPs, but we can learn from other clinical 
medical settings where e-learning appears to be at least as effec- 
tive as traditional instructor-led methods such as lectures. An 
e-learning course can also be integrated in 'blended learning' - 
a combination of technology-based materials and face-to-face 
sessions used together. In addition, by offering blended learning 
to OPs, a maximum of learning approach preferences can be 
met in this way, resulting in better learning outcomes [32,33]. 

Since WWW is becoming more and more accessible for 
OPs all over the world, it is feasible to promote e-learning as a 
learning tool in occupational health care. E-learning technolo- 
gies offer learners the control over learning sequence, pace 
of learning, time, and often media. Furthermore, e-learning 
offers educational centres the opportunity to reach OPs who 
are spread over a wide geographical area. On a larger scale, 
international collaboration can be achieved to develop more 
e-learning courses for OPs and other occupational health pro- 
fessionals, and evaluate the effectiveness of these courses on a 
larger scale. 

In particular, since EBM is relatively new in the occupa- 
tional health care field, an online introductory course can be 
useful for enhancing at least the EBM knowledge of OPs. For 
improving the EBM skills and EBM behaviour of OPs, more 
intensive education and training will be needed. 

In conclusion, our electronic introductory course on EBM 



was applicable and successful in increasing OPs' EBM knowl- 
edge and self-efficacy in practising EBM. An improvement 
in EBM skills was found directly after the course, but did not 
sustain over time. The course was not effective in changing the 
already high attitude of OPs, the social influence, and EBM 
behaviour. It seems that e-learning courses can be integrated 
with blended learning, where it can be used next to other edu- 
cational methods that are effective in changing behaviour. 
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APPENDIX. 



Subjects covered in the seven topics of the electronic EBM course 

1 . Basic principles of the evidence-based medicine (EBM) (23 screens) 

What is EBM? 

Why is EBM important? 

What are the different levels of 'evidence'? 

What phases does the EBM process involve? 

2. Types of questions (26 screens) 

What are health, legal, and statistical questions in practice? 

What are background and foreground questions? 

Four types of health questions 

Assignment: categorizing questions in practice 

3. The patient, intervention, comparison, outcome (PICO) method (10 screens) 

What is the PICO system? 

What is the significance of the PICO system 

How does the PICO system work? 

Assignment: reformulating a case study in PICO terms 

4. Information sources and search strategies (20 screens) 

What is efficient and reliable searching? 
Text books and manuals 
Occupational Medicine 
International journals 
Websites on internet 
Clinical practice guidelines 
Databases 
Search sequence 

Assignment: link between question and information source 

5. Searching in MEDLINE with PubMed (29 screens) 

What is MEDLINE and what is PubMed? 

How do I use search terms? 

How do I translate search terms into MeSH terms? 

How can I use Boolean operators? 

What are search filters? (+ an example) 

Assignment: performing a PubMed search 

6. Evaluating results (41 screens) 

What is the meaning of 'statistically significant'? 
Results of etiological and prognostic research 
Results of diagnostic research 
What are reviews? 

Critical appraisal (valid, reliable, applicable) 
Assignment: quality evaluation 

7. Answering questions (1 5 screens) 

What is a critically appraised topic (CAT)? 

Some examples of CATs 

Is the result applicable in practice? 
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